1. Field of the Invention
The present invention relates to a technique for calculating the positions and orientations of indices allocated on a physical space.
2. Description of the Related Art
In recent years, studies about Mixed Reality (MR) techniques have been extensively made. The MR techniques aim at seamlessly merging a physical space and a virtual space created by a computer. Of the MR techniques, an Augmented Reality (AR) technique has especially received a lot of attention.
An image display apparatus based on the AR technique is implemented by a video see-through system or optical see-through system. The video see-through system displays a composite image obtained by superposing and rendering, on an image of a physical space captured by an image capturing device such as a video camera or the like, an image of a virtual space generated in accordance with the position and orientation of this image capturing device. Note that the image of the virtual space includes a virtual object, text information, and the like rendered by computer graphics. The optical see-through system displays, on an optical see-through display mounted on the head of an observer, an image of a virtual space generated according to the position and orientation of the viewpoint of the observer.
Applications of the AR technique to various fields are plausible including an operation assistance that superimposes the conditions in the body onto the body surface of a patient undergoing a surgery, an architectural simulation that superimposes a virtual building on a vacant lot, an assembly assistance that superimposes the work sequence and the wiring state upon assembly, and the like.
One of the most important problems in the AR technique is how accurately the physical space and virtual space are registered, and many conventional efforts have been done. The registration problem in AR amounts to the problem of calculating the position and orientation of an image capturing device in a scene (i.e., on a reference coordinate system defined in the scene) in case of the video see-through system. Likewise, that problem amounts to the problem of calculating the position and orientation of the viewpoint of the observer or display in a scene in case of the optical see-through system.
As a method of solving the former problem, the following method is generally adopted. That is, a plurality of indices are allocated or set in a scene, and the position and orientation of an image capturing device on the reference coordinate system are calculated based on the correspondence between the projected positions of the indices in an image captured by the image capturing device, and the positions of the indices as given information on the reference coordinate system. As a method of solving the latter problem, the following method is generally used. That is, it is a common practice to mount an image capturing device on an object to be measured (i.e., on the head of the observer or on a display), and to calculate the position and orientation of the image capturing device by the same method as in the former method, thereby calculating the position and orientation of the object to be measured based on the calculation result.
The method of calculating the position and orientation of the image capturing device based on the correspondence between the projected images of indices on an image and the three-dimensional (3D) positions of these indices have been conventionally proposed in the field of photogrammetry (see non-patent reference 1).
Non-patent reference 2 proposes the following technique. That is, the position and orientation of the image capturing device, which are calculated based on the projected images of the indices on the image, are set as initial values, and the position and orientation of the image capturing device are optimized by iterative calculations so as to minimize errors between the actual observation positions of the projected images of the indices on the image, and the calculated positions of the projected images. Note that the calculated positions of the projected images are calculated from the 3D positions of the indices and the position and orientation of the image capturing device.
With the aforementioned conventional method, the position and orientation of the image capturing device are calculated based on the image captured by the image capturing device.
On the other hand, methods disclosed in, e.g., patent references 1 and 2 and non-patent reference 3 have been conventionally executed. That is, a six-degree-of-freedom position and orientation sensor such as a magnetic sensor, ultrasonic sensor, or the like is attached to an image capturing device as an object to be measured, and the position and orientation are measured in combination with detection of indices by the aforementioned image processing. Since the sensor output value can be stably obtained although its precision varies depending on the measurement range, the method that uses the sensor and image processing together can improve robustness compared to the method using the image processing alone.
In patent reference 2, the position and orientation of the image capturing device obtained from the six-degree-of-freedom position and orientation sensor are set as initial values, and the position and orientation of the image capturing device are calculated by minimizing errors between the observation positions and calculated positions of the projected images of the indices on the image by iterative calculations.
In the aforementioned registration method using indices, the following pieces of information need to be given in order to calculate the position and orientation of the image capturing device as an object to be measured on the reference coordinate system. That is, in case of dot-shaped indices (dot indices) like circular regions with an identical color on a space, the positions of their barycenters on the reference coordinate system, and the orientations of the indices with respect to the reference coordinate system need to be given. In case of polygonal indices such as triangular indices, square indices, and the like, the positions of their barycenters on the reference coordinate system need to be given. In case of such polygonal index such as a square index and the like, the index itself may often be used as a reference for a coordinate system without defining any reference coordinate system. However, when a plurality of indices are used, since the relationship between their positions and orientations need to be given, a reference coordinate system is required.
The position and orientation of each index can be measured manually using a surveying tape or protractor, or using a surveying instrument. However, measurements using images are made in terms of the precision and labor. The position of a dot index can be measured by a method called a bundle adjustment method.
The bundle adjustment method uses the following technique. That is, a large number of images of a dot index are captured by the image capturing device, and the observation positions of projected images of the point indices on the images are calculated. The position of the dot index and the position and orientation of the image capturing device are repetitively corrected by iterative calculations so as to minimize errors (projection errors) between the calculated observation positions, and the calculated positions of projected images of the dot index, which are calculated based on the 3D position of the dot index and the position and orientation of the image capturing device. In this way, the position of the dot index is calculated.
Non-patent reference 4 discloses a method of measuring the positions and orientations of a large number of square indices allocated on a 3D space. In non-patent reference 4, a large number of images of the large number of square indices allocated on the 3D space are captured, and the position and orientation of the image capturing device which captured the images, and the positions and orientations of the square indices are calculated by iterative calculations so as to minimize projection errors.
In order to implement the MR and AR, the position and orientation of the image capturing device need to be measured. As one technique for measuring the position and orientation of the image capturing device, a method of capturing an index whose position on the physical space is given by the image capturing device, and using the captured image is known.
However, upon allocating the “index whose position on the physical space is given” on the physical space, its position need to be measured, but that measurement is troublesome. Hence, an “index calibration tool” which calculates the position of this index using a plurality of images obtained by capturing the index allocated on the physical space using a plurality of image capturing devices has been proposed (see non-patent reference 5). Using such index calibration tool, the position of the index on the physical space can be easily measured.    [Patent Reference 1]    Japanese Patent Laid-Open No. 11-084307    [Patent Reference 2]    Japanese Patent Laid-Open No. 2000-041173    [Non-patent Reference 1]    R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, vol. 13, no. 3, pp. 331-356, 1994.    [Non-patent Reference 2]    Kato, M. Billinghurst, Asano, and Tachibana: “An Augmented Reality System and its Calibration based on Marker Tracking”, Transactions of the Virtual Reality Society of Japan, vol. 4, no. 4, pp. 607-616, 1999.    [Non-patent Reference 3]    A. State, G. Hirota, D. T. Chen, W. F. Garrett and M. A. Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH'96, pp. 429-438, 1996.    [Non-patent Reference 4]    G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multi-marker calibration for augmented reality applications”, Proc. ISMAR2002, pp. 107-116, 2002.    [Non-patent Reference 5]    D. Kotake, S. Uchiyama, and H. Yamamoto: “A marker calibration method utilizing a priori knowledge on marker arrangement,” Proc. 3rd IEEE/ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2004), pp. 89-98, November 2004.
With the dot index position measurement method using the aforementioned bundle adjustment method, the accurate position and orientation cannot be calculated unless at least two images are captured per index. In case of a dot index, even if only one image is captured, a straight line where the dot index exists on the physical space is merely determined, and the position of the dot index in the depth direction of the image capturing device becomes indefinite. For this reason, information indicating whether or not minimum required images for the position measurement have been acquired is important.
Upon capturing an image, information indicating whether each individual index is a reference index or an index to be calibrated is important.
As a result of calibration, a specific index cannot often be calibrated. In such case, calculation errors cannot often be eliminated unless an image that disturbs the calculations is removed, and information indicating which image disturbs the calculations is required.
Upon execution of calibration again by adding image data after calibration, information indicating which indices have been calibrated is required.
In case of a polygonal index such as a square index, the index itself has a unique ID. Even when this index is designated in advance as an object to be calibrated, it can be automatically added as an object to be calibrated when it is recognized from an image. However, since the automatically added index may be an index which is not to be added as an object to be calibrated, information indicating whether or not the automatically added index is an index designated in advance is required.
Since errors are distributed to a large number of images with increasing number of images including indices, the position and orientation can be obtained with higher accuracy than a case of a smaller number of images. For this reason, it is desirable to capture an image of an index so as to increase the number of times of capturing of the index whose position and orientation are to be calculated as much as possible. For this reason, information indicating the number of times of capturing of each index need to be presented. However, upon acquisition of images for calibration, various conditions need to be considered, and it is difficult for the operator to judge what to do next by combining these conditions.
In order to efficiently acquire the positions of indices on the physical space using the index calibration tool, all indices need to be evenly captured. For this purpose, position information need to be equally stored for respective indices.
However, in the conventional system, information indicating which index has already been captured is not provided. For this reason, the user need to remember about whether or not each individual index has already been captured upon capturing indices. If the user forgets to capture a given index, the user need to capture that index, and troublesome operations are required to calculate the positions of indices.